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ABSTRACT 



This report includes comments and recommendations from a 



group of consultant-evaluators at the North Central Accreditation Commission 
(NCA) , assessing different student learning evaluation techniques utilized by 
440 higher education institutions, including 162 two-year colleges. 
Recommendations include linking the assessment of student learning with other 
institutional effectiveness evaluations. Faculty participation in evaluation 
efforts is viewed as a necessity by NCA consultants. The hiring of an 
assessment administrator is also recommended in the report. The report offers 
methodology and measurement suggestions for assessing student learning, 
including subject matter pre- and post-tests, random sampling of classes to 
ensure reliability and validity, national standardized tests, student exit 
interviews, alumni surveys, employer surveys, transfer rates, and job 
placement data. NCA consultants report that institutions may assess students’ 
cognitive, behavioral, and/or affective learning. The report provides a list 
of the traits present in higher education institutions with strong commitment 
and support for assessing student learning; for example, strong governing 
board support, campus- wide data collection office and system, and many 
published reports of student learning assessments. A list of traits present 
at institutions with weak commitment and support is also provided in the 
report. (MKF) 
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from the original document. 
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Note from the Commission on the 1997 Reprint 

This document was first made available last year at the 1996 Annual Meeting. Dr. Lopez had been using 
and refining a typescript edition for several months before that meeting, and some who had read it 
encouraged its printing for broader distribution. A quick reading of the Collection of Papers for the 1997 
Meeting suggests that it has been a useful resource to several institutions and Consultant-Evaluators. 
Therefore, the Commission reprints it for your use and consideration. 

After careful consideration, Dr. Lopez has decided not to update or edit the paper. In 1993-1995 most 
Consultant-Evaluators and the institutions they visited were at best just beginning to move from plan- 
ning assessment programs to implementing them. Nonetheless, the advice provide in the document, 
particularly the observations and recommendations in the shaded portions, continues to be useful, 
However, Dr. Lopez rightly concluded that the next generation of literature about assessment ought not 
to be a revised, updated version of this paper; instead, it should be rich with examples and case studies 
drawn from our institutions’ ongoing experiences with the assessment of student learning. 

Many of the formal presentations on assessment at the 1997 Annual Meeting will become the basis for 
that new literature. The implementation of assessment programs in the Commission’s highly diverse 
membership highlights both those lessons that cut across all types of institutions and those that might 
well be applicable only to institutions of specific scope, size, and mission. In responding to the 
Commission’s Assessment Initiative many institutions discover that they are engaged in studying, if not 
changing, their basic educational cultures. This can be slow work. Such changes in the future most likely 
will be shaped by good practices and by institutional experiences with effectively conceived and well- 
implemented programs to assess student academic achievement. 
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Opportunities for Improvement: 

Advice From Consultant-Evaluators on 
Assessing^ Student Learnings 



I n 1989 the NCA Commission approved an assessment initiative that as a first step called for its member 
institutions to submit plans for assessing the learning of their students at every level of study. By March 1, 
1996, the assessment plans of a majority of member institutions had been reviewed by Consultant-Evaluators 
serving either on Assessment Plan Review (APR) panels or on on-site evaluation teams. Now that their plans 
have been approved through one of two means of peer review, institutions are implementing their proposed 
assessment programs. 

As staff reviewed the comments and suggestions these groups of Evaluators had provided individual institutions 
on assessing student academic achievement and on using the results to improve student learning, patterns of 
characteristics considered to be strengths of or to raise concerns about the plans and programs began to emerge. 
It became clear that by analyzing a sample of the reports it would be possible to determine what had been 
repeatedly praised or constructively criticized during the peer reviews. Staff agreed that this information could 
be useful to all member institutions during the initial stages of implementation and subsequent development of 
their respective programs to assess student academic achievement and improve student learning. 

The purpose of this paper is to provide an overview of the positive comments cmd useful advice Consultant- 
Evaluators made to several hundred NCA member institutions in Team Reports and APR forms over the last 
three academic years, 1993-94 to 1995-96. Staff hope you will find it helpful in sustaining or heightening the 
energy your campus brings to its assessment of student academic achievement and its creative application of 
the results of assessment to efforts to improve student learning. The study should be read with recognition that 
the peer Consultant-Evaluator judgments were made in light of what they accept as good professional practice in 
measuring student academic achievement and in using those results to make any changes needed to improve 
student learning. 



Therefore, the findings are to be understood as being the constructive advice of peers; they should 
not be construed as Commission policy. 



The Sample 

The comments and advice that follow are based on an analysis of Team Reports from 120 comprehensive visits that 
occurred in 1993-94, 109 comprehensive visits in 1994-95, and on all of the assessment plcm reviews carried out by 
APR panelists between January 1 cind October 30, 1995. This paper therefore summarizes the comments of Consult- 
ant-Evaluators on assessment plcins and programs made to approximately 45% (440 of 981) of all member institu- 
tions. In this semnple of 440 institutions are 45.7% (162 of 354) of all 2-year colleges; 47.7% (95 of 199) of the institutions 
offering only the baccalaureate; 43% (103 of 239) of those that offer master degrees; 24% (5 of 21) of the institutions 
that offer the specialist degree, and 44.6% (75 of 168) of all doctoral degree-granting universities. 



The Analysis of Team Reports and APR Reviews 

The written comments from Team Reports and APR forms were coded and analyzed using the constant compara- 
tive method of qualitative research (Glaser & Strauss, 1967; Marshall & Rossman, 1989; Strauss & Corbin, 1990). 
Verbatim quotations from each Team Report and APR form were recorded, then coded and compared by subject 
content until dominant themes emerged. The subject headings that appear below under Section 111, The Findings, 
represent the subject themes for which patterns emerged and were repeated by a majority of the Evaluators. 
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The Findings 

A high level of internal consistency appears among Evaluators’ views as to what characteristics of assessment 
plans and programs are considered strengths and what are causes for concern in the written comments and 
advice offered institutions of every type and level by both Evaluation Teams and APR panels. This finding is 
significant. It suggests that regardless of the year in which an evaluation was conducted and whether the evalu- 
ation was made by peer Consultant-Evaluators serving as Team members on a Comprehensive Visit or by those 
serving as APR reviewers reading independently, Evaluators were guided in their judgments by a common under- 
standing of what does and does not constitute good practice in the assessment of academic achievement and the 
use of its results. 

What follows is a summary of those complimentary and critical comments and suggestions for improvement 
found frequently repeated throughout the Team Reports and the forms submitted by APR panelists. Since no 
substantive difference was found in the comments made by Evaluators on the plans submitted by 2-year, 4-year, 
or graduate institutions, private or public, the observations made under each heading are relevant, and there- 
fore can be useful, to any institution engaged in assessing student academic achievement, regardless of its type 
or level. 



A. Mission and Goals 

The assertion that institutions must link departmental and programmatic objectives for student learning 
with Institutional mission and goals is stated as an imperative by Team members and APR panelists in their 
evaluations of the assessment plans submitted by all types of institutions. Evaluators commend the assess- 
ment programs of institutions in which academic departments and programs have clearly linked their as- 
sessment activities to their own statements of purpose and goals, and to their objectives for student learn- 
ing, and in which all of these are reflective of relevant portions of the institution’s Mission and Goals state- 
ment and its published educational purposes. 



B. The Relation Between Assessing Student Academic Achievement and Evaluating Other 
Aspects of Institutional Effectiveness 

Evaluators stress the importance of institutions distinguishing between assessing student academic achieve- 
ment and evaluating overall institutional effectiveness. In their early efforts to develop a program for as- 
sessing student learning, many institutions found it difficult to separate the evaluation of what and how 
much students have learned from the evaluation of those parts of the college or university that enable the 
students to learn. Although all institutions of higher education exist to educate students, they frequently 
have other secondary purposes and carry on a broad range of activities to accomplish both educational 
and non-educational goals. Measuring institutional effectiveness, therefore, requires that each institution 
first agree upon what it will consider components of institutional effectiveness and then develop an assess- 
ment program for each of those components. The program for assessing student academic achievement 
must be distinct and able to stand independent of interrelated and equally valuable programs for the evalu- 
ation of components of institutional effectiveness other than student learning (such as providing resources 
and services needed to meet the educational needs of students). 

Team members and APR panelists repeatedly comment on the unfortunate effects of an institution’s at- 
tempting to apply to the assessment of student learning, goals and processes appropriate to evaluating 
other components of institutional effectiveness but not to the measurement of student academic achieve- 
ment. In sum, in order to establish strong patterns of evidence for Criterion Three, an institution’s program 
for the assessment of student academic achievement should be an independent, clearly identifiable com- 
ponent of any more comprehensive program to evaluate overall institutional effectiveness. 
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C. Faculty Participation in Assessment of Student Learning 

An important characteristic of an effective assessment program is that it be “faculty owned and driven” 
(i.e., that evidence be provided of active and on-going faculty participation in the planning and implementa- 
tion of the program to assess student learning). Evaluators stress the necessity for the engagement of the 
on-campus and off-campus faculty of every academic unit in the development and operation of the assess- 
ment program. They also emphasize that the individuals or groups responsible for the assessment program 
should respond quickly and directly to faculty fears or ambivalence about assessing student learning. 

Team members and APR panelists stress the importance of having faculty determine how information pro- 
vided by the assessment program will be used to effect needed curricular change, more effective academic 
support services, and improvement of teaching. They urge faculty as well as administrators to insist on the 
development and implementation of assessment plans for all academic programs and to participate ac- 
tively in both processes. Evaluators often advise administrators to provide incentives for faculty to partici- 
pate in assessment activities and recognition of those who do. They also emphasize the importance of 
faculty involvement in determining how assessment results will be utilized to identify what is needed to 
improve student learning and in monitoring the effects of any changes made for the purpose of enhancing 
student learning to determine whether and to what extent students’ academic achievement improves. In 
other words. Evaluators praise assessment plans and programs that are described as having emerged after 
extensive collegial discussion and those which reflect that a concern for improving student learning perme- 
ates the culture of the institution. 



What Evaluators Typically Expect of Faculty 

i> awareness of any coordinated effort within their college to assess student academic achieve- 
ment; 

i> ability to describe the key elements of the College’s assessment plan or program, that is, 
know: 

o how the assessment program functions at the institution; 
o what role faculty play in its operation; 

o what measures and standards have been proposed and adopted for assessing student 
learning; and 

o how the results of assessment are to be used to identify changes that may be needed if 
student learning is to improve in their respective programs. 



D. Linking the Results of Assessing Student Learning with Departmental and Institutional 
Planning and Budgeting 

Evaluators cite as critical the need to link the results of student academic achievement to departmental 
and institutional planning and budgeting processes. They suggest that institutions clearly define the pro- 
cess by which requests (e.g., for personnel, curricular changes, facilities, equipment) that arise from fac- 
ulty review of the results of assessment of student learning will be incorporated into the ongoing strategic 
planning and budgeting processes. They ask for information about what will be done to assure that re- 
quests most likely to improve student academic achievement are given appropriate priority among com- 
peting requests for limited resources. 
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E. Assessment Program Administration: The Chief Academic Officer 

Evaluators were unanimous about the need for assessment programs to identify an individual who will be 
accountable for overseeing assessment throughout the institution. Evaluators are clear that committees 
alone cannot provide effective leadership. Moreover, they advise that when an advisory or coordinating 
Assessment Committee is appointed it report directly to the chief academic officer (CAO). Evaluators fre- 
quently cite modification of the CAO’s job description to include responsibility for administering the pro- 
gram to assess student academic achievement as evidence of institutional commitment to on-going im- 
provement in student learning. 

In large institutions that have an office or person responsible for conducting institutional research, the full- 
time or part-time assignment of an institutional research staff member to assist the CAO with assessment is 
considered by many Evaluators an indication that the institution has in fact delegated authority and ac- 
countability to an officer charged with responsibility to supervise and take initiative in the assessment 
process and not merely “monitor” it. In small institutions without an office or officer of institutional re- 
search, other designated means of support for the CAO such as interns or faculty members assigned as 
part of load or service can accomplish the same ends. 



F. Assessment Program Administration: The Assessment Committee 

Whatever its name, Evaluators stress that the committee charged with on-going responsibility for coordi- 
nating assessment is best positioned for success when it is a joint faculty-administration standing commit- 
tee with campus-wide representation. Evaluators urge that the majority of the Assessment Committee mem- 
bership be broadly representative of the faculty of the institution’s academic units (colleges, schools, divi- 
sions, departments, and interdisciplinary programs) and include staff from the institutional research office 
and the student affairs office, as well as an academic officer. 

Evaluators agree that an Assessment Committee’s responsibilities should include the monitoring of the 
implementation of the assessment plan (i.e., developing, maintaining, evaluating, and modifying as needed 
all assessment activities). Typically, such a committee is also responsible for efforts to educate the 
institution’s constituent academic units about the significance of their on-going participation in the assess- 
ment process, to provide the units with any assistance they may need in obtaining bibliographies on as- 
sessment, and to train them in the selection and use of instruments and measures appropriate to the kinds 
of learning they want to measure. Evaluators urge assessment committees to provide ample time for aca- 
demic units to meet the obligations of assessment and to allow twice the time estimated in order to carry 
out the work involved in collecting and analyzing the information before quantitative and narrative results 
can be distributed and used by faculty. 

Some rather practical ideas from the comments of Evaluators about the breadth of the work of an Assess- 
ment Committee have emerged. Specifically, Evaluators propose that an Assessment Committee be del- 
egated responsibility for following standard operating procedures for committees, providing support to all 
aspects of the assessment process, receiving all data and materials generated by assessment activities, 
recommending improvements in the assessment program, and disseminating reports on the results of as- 
sessment and the initiatives based on assessment intended to improve student leaning. Each of these re- 
sponsibilities is summarized below. 



Provide Support 

i) follow good operating practices for any committee such as publicize meetings and main- 
tain and distribute minutes of every meeting; 

0 ensure that the assessment procedures and practices adopted by the Assessment Com- 
mittee, faculty, and administration are professionally sound and meet the needs of the 
institution; 
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l> distribute throughout the institution the names and titles of the persons and offices charged 
with collecting, analyzing, and disseminating assessment data, and also the process, play- 
ers, and calendar to be used in acting upon assessment data. 

i> develop, distribute, and monitor a schedule pertaining to assessment (i.e., an assessment 
calendar), listing the specific dates on or by which 

o assessment instruments will be evaluated; 

o each kind of data and analysis will be submitted to the Committee; 

o the Committee will meet to review these data and analyses; 

o the Committee’s summary and commentary will be transferred to a committee 
that has responsibility for curriculum. 

i> develop a Source Book containing material to help academic units develop or adopt ap- 
propriate methods for gathering the kinds of information they will need for assessment 
and for measuring cognitive, behavioral, and affective learning. 



Review 

i) compile assessment results (numerical data and narrative analysis) from the institutional 
research office or from the academic units; 

i) compile suggestions or recommendations from academic units for changes to introduce 
for the purpose of improving student learning; 

i) compile assessment results received from academic units made after recommended 
changes have been introduced and compare them with pre-change assessment results; 

i) summarize and interpret results in a written progress report to the CAO, President, and 
Board at the end of each semester. 



Recommend 

i> identify, prioritize, and recommend those resources needed for the coming year to main- 
tain or improve the academic assessment program that transcend or cannot be accom- 
plished within a given school or department; 

i> make recommendations to the CAO for: 

o additional assessment processes and modifications in present practices and/ 
or process; 

o modifications in the assessment process and calendar; 

o integrating assessment and evaluation into the planning and budgeting cal- 
endar and processes at each level (institutional, college, school, division, 
departmental). 
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Report 

0 have frequent, regularly issued progress reports prepared and distributed to internal con- 
stituencies; 

0 publish an annual report describing the year’s outcomes of the assessment of student 
academic achievement; 

0 issue a report card to each academic unit at the end of each term summarizing the degree 
to which students are meeting their faculty’s educational objectives. 



G. Assessment Program Administration in Decentralized Institutions 

In highly complex, decentralized universities, Evaluators find that assessment programs operated wholly 
through the central administration often result in the faculties of the various colleges and schools of the 
university being either uninformed about the results of assessment efforts, or, if informed, skeptical that 
the results have credibility or are relevant to students in their respective units. Evaluators therefore sug- 
gest that the Administration establish a structure appropriate to the institution’s size and composition to 
plan and implement the assessment of student learning across academic units. Evaluators not only recom- 
mend establishing an Assessment Steering Committee, but they also urge that an assessment coordinator 
be appointed with full or substantial release time to handle the multiple demands of an assessment pro- 
gram in a large, complex institution. 

Evaluators find it helpful in this type of institution for the Assessment Steering Committee and Assess- 
ment Coordinator to be assisted by college-level (or school-level) assessment coordinators, and for there 
to be an assessment coordinator exclusively for the General Education program as well. The relationships 
among these individuals and groups, and between them, the administration, and the Faculty Senate, need 
to be clearly stated in a document that describes the student academic achievement assessment pro- 
cesses and activities. This assessment document needs to explain how the steps and timetable in the 
Assessment Program fit into the ongoing planning and budgeting processes and timetable of the academic 
units and of the institution as a whole. The document should be widely distributed and discussed through- 
out the Board, Central Administration, and academic units so that all parties understand participants’ 
respective responsibilities and authority and can begin to integrate assessment activities into the aca- 
demic life of the institution. 



H. Assessment of Student Learning in General Education 

Many institutions are using the Commission’s assessment initiative as the impetus for revisiting their gen- 
eral education philosophy and objectives and their general education requirements and courses. Evalua- 
tors agree that such a reconsideration of the general education program provides a valuable starting place 
for institutions preparing to undertake or revisit assessment of student academic achievement. They com- 
mend institutions that after reviewing their genercd education program, demonstrate a renewed emphasis 
in the program on critical thinking and problem-solving skills. 

Evaluators frequently assert that because general education involves participation of departments univer- 
sity-wide, the development of a structure and timetable for the assessment of the student learning that 
occurs between the beginning and completion of the Genercd Education program needs the same careful 
attention required when establishing assessment in a decentralized university. To be effective, assessment 
of general education, like assessment across schools of a university, typically requires the appointment of 
an individual with personal commitment and adequate time and staff support to carry out the responsibili- 
ties of coordination and leadership of the endeavor. If the General Education program does not presently 
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have its own written statement of purposes and objectives that flow from the institution’s mission and 
goals statement, these should be drafted and agreed upon before a plan or program to assess student 
learning in general education is undertaken. If the objectives for general education are clearly stated and 
measurable, appropriate means for assessing the students’ academic achievement can easily be found for 
each objective. 



I. Assessment at the Level of the Academic Unit (College or School, Division, Department, Inter- 
disciplinary Program, General Education Program, Major, or Graduate Degree Program) 

Evaluators praise academic units’ assessment plans and programs when the unit uses its own statement of 
purpose and educational objectives to frame its statements about the characteristics and competencies it 
intends its students to acquire in a given major or graduate degree program. They find it important for 
faculty to establish objectives for what they expect their students to learn (i.e., for faculty to define measur- 
able learner outcomes). By measuring how much students actually do learn, faculty can then determine 
whether there is a gap, and if so how large a gap between their intentions and students’ achievement. When 
faculty expectations and student performance are at odds, faculty need to determine the reasons for the 
discrepancy. Academic units can then make adjustments in curriculum, institutional delivery, etc., and test 
to determine which of those changes correlate with measurable improvements in student learning. 

Evaluators stress that to accomplish the purposes of assessment, data obtained and analyzed by institu- 
tional offices or personnel need to be routinely shared with all academic units. Once faculty and academic 
administrators have agreed to modify specific aspects of personnel, curriculum, modes of instruction, li- 
brary holdings, other academic support, equipment and facilities (to see if the change will improve student 
learning), their recommendations should be incorporated into the long range and annual departmental 
planning documents and budget requests. Evaluators note that it is important for departments to docu- 
ment the steps in the process and timetable they will follow to test and evaluate changes. 

/. The Link Between Assessment of Student Academic Achievement and Evaluation of Depart- 
ments and Programs 

Evaluators caution that neither evaluations of teaching and other aspects of faculty performance by 
peers and students, nor internal and external peer reviews of departments and programs (in cases 
where such reviews do not include assessment of student learning as a component) are appropriate 
to use as measures of what students have learned in an undergraduate major or graduate degree 
program. In contrast, they find that the results of assessing student achievement in general education 
and in major or graduate degree programs over time do provide one of several valuable pieces of 
information to be considered when a department or program is being evaluated. Evaluators advise 
institutions to include in their written Program Review process one or more steps in which informa- 
tion generated by student academic assessment over several successive years will be included in 
Program Review. Linking assessment of student learning with academic program review in this way is 
efficient and economical. Evaluators find this integration to be a reflection of the institution’s commit- 
ment both to continuous improvement of academic achievement and to strengthening academic pro- 
grams at both the undergraduate and graduate levels. 

2. The Role of Specialized Accreditation in Assessing Undergraduate and Graduate Student Learning 

Teams find that undergraduate majors and graduate degree programs within departments or divi- 
sions accredited by specialized (discipline-specific) accreditation bodies typically have better devel- 
oped and more comprehensive assessment plans or programs than other departments. However, APR 
panelists caution academic units not to rely upon specialized accreditation to provide the only incen- 
tive and means of assessing student learning or to assume that specialized accreditation is a guaran- 
tee that an assessment process is in place. Some specialized accrediting bodies measure a program’s 
compliance with input standards but do not require that learner outcomes be compared with informa- 
tion on entrance knowledge; they are therefore not measuring the educational results of the program 
at all. 
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J. Providing for Communication: Feedback Loops and Reports 

Comments of the Evaluators indicate that two forms of written communication exist in effective Assess- 
ment Programs: feedback loops and reporting. Feedback loops provide those who generate data and or 
make recommendations with timely, accurate accounts of the use the recipients have made of information 
and the action they have taken on recommendations. Examples the Evaluators cite include (1) an Assess- 
ment Committee or institutional research officer sends the analysis and interpretation of the assessment 
testing scores received from an academic unit back to that department or school; (2) an academic unit 
forwards its actions based on that information to the Assessment Committee and/or the committees charged 
with curriculum and program review. 

In the second type of communication, reporting, an individual (e.g. CAO, department chair, or program 
director) or group (Assessment Committee, Public Relations Office, President’s Office) distributes concise 
informational documents that present findings, interpretation of data, actions based on analyses of data, 
and the results of those actions to appropriate internal and external constituents. The purpose of this form 
of communication is to promote ownership of and support for the assessment effort throughout the institu- 
tional community. 

Evaluators find that when adequate provisions for ongoing communication are in place, the results of as- 
sessing student academic achievement become the basis for faculty and academic administrators to agree 
upon changes that could improve student learning and are therefore important to introduce. Such changes 
are then used to update planning documents, and to include as proposed priority expenditures in annual 
budgets. For this reason Team members regularly ask institutions to describe the process by which the 
changes that faculty and academic administrators wish to make to improve student learning (after study- 
ing assessment data) are incorporated into the regular planning and budgeting processes of the depart- 
ment, the academic unit to which it belongs, and the institution. 

/. Feedback Loops 

In speaking of intra-institutional communication in the assessment process. Evaluators praise assess- 
ment programs that have “smooth and seamless feedback loops.” They also call for institutions to 
define in the assessment program document at what point and by what means faculty will be autho- 
rized to introduce changes they believe that the assessment results suggest it is important to try in 
order to improve student learning, if such changes do not require expenditures greater than the ap- 
proved departmental programmatic budget allows. 

2. Reports 

As indicated in the description of the recommended duties of the Assessment Committee (pp. 6-8), 
after data are collected, analyzed and reported to deans and faculty by the office or individual charged 
with those tasks, the Assessment Committee should arrange for the findings, and subsequent actions 
taken in response to the findings, to be fully and promptly communicated to all constituencies of the 
institutional community. Evaluators suggest the inclusion of three kinds of reports in the assessment 
program that can be an effective means of communication: Progress Reports, the Annual Report, and 
the Report Card. 

□ Progress Reports: Wide distribution of Progress Reports that publish the results of assessment 
and the changes being introduced based on that information by academic units to improve stu- 
dent learning were noted as one important means of keeping all parts of the institution aware of 
assessment activities. Regular, frequent Progress Reports can be a morale booster to faculty in 
the departments and programs as they engage in assessment because the results of their efforts 
to improve or maintain a high level of student learning are quickly brought to their attention. 
Indirectly, they also communicate to all constituencies the importance attached by the Board 
and top administration to useful and timely feedback in assessment, decision-making, planning, 
and budgeting. 



Cecilia L Lopez 



Staff Paper 



March 1 996; Reprinted April 1997 



Opportunities for Improvement: Advice from Consultont-Evoluotors on Assessing Student Learning 



II 



□ Annual Assessment Report: Evaluators find that a second valuable publication is an Annual Re- 
port on Student Academic Achievement. Like Progress Reports, the Annual Report can be an 
effective means of enhancing communication about assessment efforts across the entire institu- 
tion. It brings the contents of the more frequent, brief Progress Reports into a single overview of 
student academic achievement and describes the effects of utilizing assessment results to im- 
prove learning throughout the institution. 



Characteristics of an Annual Assessment Report 

i) includes both assessment results and responses to those assessment findings 
for all educational programs in the institution: General Education program, ma- 
jors and minors within undergraduate departments and interdisciplinary units, 
and graduate degree programs within graduate colleges and schools; 

i) presents to the community these first findings for the educational programs 
within the institution together with an interpretation of their meaning; 

i> alerts readers to changes in educational objectives and measures for the vari- 
ous programs adopted by the departments and other academic units and de- 
scribes the actions each has decided to take to test whether certain proposed 
changes will in fact improve student learning; 

i> informs the community on which changes introduced appear to be directly re- 
lated to improved student academic achievement and which did not. 



Evaluators recommend that the Annual Report be distributed to all of the institution’s constitu- 
encies, including the student body and alumni, and note that the CEO and Board members often 
find it useful in public relations and fund raising activities. 

□ Report Card: A third type of report suggested by a number of Evaluators is the Report Card. The 
Assessment Committee, after each term or at the end of the academic year, issues a two-part 
Report Card providing information to all academic units. Part one provides information on how 
well their students appear to be meeting the faculty’s objectives for academic achievement. Part 
two offers suggestions for how faculty might improve the assessment process being used in the 
academic unit (e.g., better measures, methodology, or more effective means for the dissemina- 
tion and utilization of results). 



K. Informing and Engaging Students in the Assessment of Academic Achievement 

Evaluators praise assessment programs that inform students about the importance of assessment (e.g., 
through descriptive material in the catalog) and that inform both students and alumni in college bulletins, 
letters, newspapers, and other published materials about the results of tests or surveys in which they have 
participated. The education and participation of student leaders through membership on assessment com- 
mittees is also regularly noted as a strength in the assessment process. 

L. Methods and Measures 
7. Terminology 

The term. Methods, as used in this section refers to the sequence of steps, or process, followed in 
order to collect data. While there are a variety of methods that can be employed to study the aca- 
demic achievement of students across a curriculum, one method, pre-and post testing, was most 
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often noted by the Consultants. The term, Measures, refers to the specific data collection instruments 
typically applied to a sample of subjects or observations. 

2. Pre- and Post-Testing 

Evaluators note that the measurement of value-added is neglected in concept and practice in many 
departmental assessment programs. For both graduate and undergraduate programs, results of pre- 
testing and post-testing were cited by Evciluators as particularly useful benchmarks by which to mea- 
sure learning from entry to exit, and thereby to measure “value added.” 

Pre-testing is not necessary if one is highly confident that students know little or none of the content 
they are to master through completing the degree program (e.g., when students enter the study of 
medical technology). In most content areas, however, pre-testing is useful in obtaining the baseline 
data needed. Pre- and post-testing are effective for measuring cognitive learning, as well as for affec- 
tive and behavioral learning assessment at the undergraduate level. Pre- and post-testing are espe- 
cially valuable for use with undergraduate transfer students in the major programs and for students in 
all kinds of graduate level programs since in both situations faculty need to determine the skills and 
knowledge the entering student brings to a program from his/her previous study. This is as important 
as discovering what an entering freshman knows about general education or what a continuing stu- 
dent of the university knows at the point of entry into the major program. In general, faculty should 
not assume that the new student, whether transfer or graduate, enters a program knowing little or 
nothing. Only by establishing a benchmark that documents exactly what the transfer or graduate 
student brings to his studies in the program by way of knowledge, skills, or behaviors, can the faculty 
later measure the growth in the cognitive, behavioral, and affective domains that has occurred over 
the course of completing the major or graduate degree. 

3. Methodological Problems 

Evaluators are concerned when the description of methods in an institution’s assessment program 
fails to specify how and when data will be collected, interpreted, and utilized, by whom, and for what 
end. They also identify other frequently encountered methodological problems such as the following: 



i> inappropriate sampling; 

i> lack of a base line against which to assess growth and development in general educa- 
tion, the major, and in graduate and professional programs; 

i> failure to determine if measures are reliable; 

i> failure to determine the adequacy of measurement procedures; 

0 systematic bias due to reliance on one measure; 

0 a lack of a system to assure that instruments have content validity (i.e., measure 
accurately what they are designed to measure). 

No Evaluator preference for quantitative over qualitative measures has been noted. Evaluators rec- 
ommend that whatever measures are used, they be continually evaluated for their appropriateness 
and “fit” in measuring the content and extent of student learning the faculty has intended. 

Finding ways to document student learning at points throughout the educational experience, not just 
before and after it, is recognized as one of the most difficult challenges in assessment, but one of the 
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most important to achieve. Therefore, practices that gather, accrue, and finally assess the cumulative 
evidence of the academic experience throughout the undergraduate, graduate, and professional pro- 
gram are cited as exemplary means of measuring learning that has taken place during completion of a 
program. Team members typically consider it to be a primary weakness if an assessment program 
lacks any specific measurement of student learning in-process (i.e., over time or at different points in 
time rather than at only the beginning or at only the completion of a program). 

4. Direct Measures of Student Learning 

Direct measures of student learning are understood to include but are not limited to: 



— the capstone experience; 



— portfolio assessment; 

— standardized tests (e.g.. Major Field Achievement Test [MFAT] in cognate areas or for Gen- 
eral Education; the Test of Critical Thinking Ability; the Academic Profile; or the Watson- 
Glaser Critical Thinking Appraisal); 

— performance on national licensure, certification or professional exams (e.g.. Professional 
Assessment Examination for Beginning Teachers [PRAXIS] or the Federal Aviation Adminis- 
tration [FAA] exam); 

— locally developed tests; 

— essay questions blind scored by faculty across the department, division, school, or college; 

— qualitative internal and external juried review of comprehensive senior projects; 

— externally reviewed exhibitions and performances in the arts; 

— external evaluation of performance during internships based on stated program objectives. 

□ The Capstone Experience. Evaluators note that the capstone experience (course, thesis, field 
project, etc.) has long been recognized as providing students with an especially effective means 
of integrating what they have learned by the end of a program. If carefully structured and docu- 
mented, it can therefore also serve as an effective assessment tool. The structure and content of 
the capstone experience should be clearly linked with the published statement of the purposes 
and educational objectives of the department or program for what their graduates are to have 
learned by completion of the program of study. Evaluators urge faculty who have decided to use 
the capstone experience as an indicator of student academic achievement to provide informa- 
tion in departmental assessment plans and in public documents available to internal constitu- 
ents, including students, about the standards they use for evaluating student learning during 
and upon completion of the capstone course or project. 



□ Portfolio Assessment. Evaluators observe that the use of student portfolios to measure student 
learning is found effective by academic units that utilize them. They urge academic departments 
that use portfolios for assessment purposes to provide evaluation protocols in departmental 
assessment program documents as to how the portfolios are to be reviewed (e.g., what the port- 
folio will include, how it will be assessed, by whom, and at what time intervals). They caution 
departments against permitting each faculty member to submit his/her own “protocol” for evalu- 
ating student work in her or his courses since that practice makes it impossible to gather and 
compare comparable data from area to area and across years. 

□ Standardized Tests. Evaluators recommend the use of multiple measures, both quantitative and 
qualitative, to assess student achievement, rather than relying solely on standardized tests. Stan- 
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dardized tests, by their very nature, tend to be generic and not well focused on specific skills or 
competencies. Evaluators counsel faculty to ask themselves whether they are utilizing a stan- 
dardized test simply because it is available or because it does in fact measure the specific learn- 
ing faculty expect of their students. If the former is the case, faculty need to search for or create 
an instrument that will provide reliable and useful information to replace the standardized test 
they are using. 

Evaluators comment on the large number of institutions that are pilot testing the use of stan- 
dardized measures in order to determine the advantages of national norming and benchmark 
comparisons with peer institutions. Examples are the Major Field Achievement Test (MEAT) in 
academic disciplines, ACT’s Collegiate Assessment of Academic Proficiency (CAAP), and ETS’ 
Academic Profile. However, they note that since standardized instruments do not measure a 
number of cognitive areas articulated in mission and purposes statements (e.g., ethically re- 
sponsible decision-making), some institutions are developing their own instruments. 

APR panelists caution against using standardized tests as assessment instruments to measure 
learning outcomes unless the particular test selected has been found to be appropriate to the 
specific learning objectives it is being used to measure. Instruments such as the Graduate Record 
Exam (GRE), the Law School Admission Test (LSAT), the Medical College Admission Test (MCAT^, 
or the Graduate Management Aptitude Test (GMAT) are nationally standardized tests that were 
originally designed, and are still utilized primarily to make decisions regarding admissions to a 
professional program. They are limited in their usefulness to measure student academic achieve- 
ment both because not all students take these tests and because they were designed to predict 
performance (i.e., the likelihood of success in a specific graduate or professional degree pro- 
gram). Additionally, Evaluators caution that standardized measures typically do not provide data 
about individual students. One example is the MFAT, which provides useful aggregate data as to 
how students majoring in a particular discipline in one institution perform as compared with 
those in comparable groups in other schools. Such instruments may provide limited information 
about individual student learning within the specific unit’s academic programs. 

□ National Licensure, Certification, or Professional Examinations. Evaluators note that scores on 
licensure or certification exams may be used as another indicator of academic achievement. 
However, Evaluators suggest that passing rates on such exams do not perse provide direct evi- 
dence of the level of achievement in the specific area of student competencies or skills tested. 
Unless licensure or certification exam scores are supplemented by information about how well 
students did in each of the subject matter areas covered in the exam, results are not likely to be 
useful to academic units that intend to use the scores as an indicator of student learning. 

In sum. Evaluators find that direct measures of student learning yield useful information about the 
value added to a student’s learning by the general education program, the major, or the graduate or 
professional program, especially when the results from multiple measures are triangulated and are 
compared with (1) baseline data and/or with (2) data from other measures taken over time. 

5. Indirect Measures of Student Learning 

Evaluators acknowledge that many sources of data if used alone are inadequate measures of student 
learning. However, some of these sources, when used to supplement direct measures, provide infor- 
mation that may enrich or illuminate aspects of what the direct measures tell us about students’ 
academic achievement. 

Evaluators consider the following as examples of indirect measures of student learning: 



alumni, employer, and student surveys; 



exit interviews of graduates and focus groups; 
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— graduate follow-up studies; 

— retention and transfer studies; 

— length of time to degree; 

— SAT scores; 

— graduation rates and transfer rates; 

— job placement data. 



M. Non-Measures 

Evaluators note that information gathered on a number of forms assumed to be “measures” of student 
academic achievement by institutions does not in fact provide evidence of learning. One such non-measure 
is a questionnaire asking students if their personal! goals for the course or major or program have been met. 
A second group of non-measures that institutions often mistakenly consider to be instruments that mea- 
sure student learning are the measures and reports associated with program evaluation. Typically they 
collect data on the quadity of curriculum and other aspects of a program. These include: 

— instruments designed for specialized program review such as the Michigan Program Review of 
Occupational Education (PROE^; 

— curriculum review reports; 

— evaluation reports of individual programs submitted by program-specific and specialized ac- 
crediting agencies, visiting committees, or committees of external peer experts. 

The third type of non-measure of student learning is information gathered not for assessment but for spe- 
cific administrative purposes: 

— faculty publications and recognition; 

— the kinds of courses or majors students select, including course enrollments and course profiles; 

— faculty/student ratios; 

— the percentage of students who study abroad; 

— enrollment trends; 

— the percentage of students who graduate with the baccalaureate in five years; 

— the diversity of the student body. 

The fourth and most frequently submitted non-measure of student learning are grades and GPAs. Evalua- 
tors regularly stress that neither grades nor GPAs are adequate or reliable measures of student learning 
across an undergraduate major or graduate/professional program of study. An excellent reference on this 
topic is by Alexander Astin, who documents in his 1 99 1 work, Assessment for Excellence, that course grades 
and GPAs “tell us little of what the student has actually learned in the course” and “very little about what a 
student actually knows or what that student’s competencies or talents really are” (p. 11). It is also worth 
noting that Pascarella and Terenzini (1991) declined to use grades or GPA as a measure of how much is 
learned during college, concluding that “there may be too many problems in the reliability and validity of 
grade point average” (p. 62). They also cite a substantial body of literature that suggests that grades may be 
influenced by a number of potential confounding factors within and across institutions (p. 63). 
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N. Measuring Student Learning Within the Three Domains 

Evaluators recommend that every academic department or other academic unit determine the extent to 
which it actually contributes to the incremental learning of its students within the three domains: cognitive 
(knowledge acquisition), behavioral (skill acquisition), or affective (attitudinal development). 

/. Cognitive Learning (Knowledge Acquisition) 

Assuming that pre-measures of knowledge and skills in the discipline have been conducted, some 
examples of direct measures of cognitive learning would be: in a General Education program pre- 
and post-testing using a measure such as the Watson-Glaser Critical Thinking Test; in a depart- 
ment of music, faculty-juried exams on studio instruction; in bachelor’s, master’s, and doctoral 
programs, comprehensive exams; and in master and doctoral programs, the oral defense of the 
thesis or dissertation. 

Evaluators note that the following do not constitute measures of cognitive learning: scores on admis- 
sion tests; progress in a master’s program; successful completion of a course; scores on a course 
exam; curriculum revisions; and commendations or recommendations received from internal or ex- 
ternal advisory committees about the curricular design, content, currency, or rigor of programs. 

2. Behavioral Learning CSkill Acquisition) 

Evaluators cite a number of observable skills typically associated with this domain including group 
interaction skills; presentation skills; oral communication skills, particularity in group contexts; lead- 
ership; interpersonal skills; managerial skills; initiative; written communication; or problem solving 
skills. Examples of direct measures of incremental behavioral learning include faculty-juried recitals 
in music or dance; assessment of actual performance in public speaking, listening, and interpersonal 
communication skills (e.g., the Communication Competency Assessment Instrument [CCAI]); and pre- 
and post- standardized measures of general skill acquisition in the context of specific content areas 
such as mathematical reasoning (e.g., Academic Profile), provided the results are made available to 
the institution for its own students. 

3. Affective Learning (Attitudinal Development) 

Consultant-Evaluators find that surveys are the most common type of measure used by institutions to 
measure the attitudinal development of their students. While Evaluators caution institutions that the 
results of student, alumni, and employer surveys are not evidence of cognitive learning, they agree 
that the information yielded by such surveys is useful in determining change and growth in what 
students have gained in the affective domain. The Cooperative Institutional Research Program ques- 
tionnaire (CIRP), which measures attitudes and opinions of students entering graduate programs, and 
the Student Goals Exploration Survey are cited as typical examples. 

□ Student Satisfaction/Attitude Surveys. Evaluators note that student-satisfaction surveys are of 
little or no value in assessing the incremental learning that a student has acquired during an 
undergraduate or graduate program. However, pre-and post-surveys that measure changes in 
attitudes towards values or beliefs yield important information about the attitudinal develop- 
ment of students. 

□ Alumni Surveys. Evaluators suggest that when an institution wants to measure the persistence of 
attitudinal changes in values or beliefs after graduation, it survey graduating students and alumni, 
in addition to utilizing other measures of growth in the affective domain. Key items on the gradu- 
ating student survey could be duplicated on the alumni survey to allow for a comparison and an 
analysis of results. This would enable the institution to measure changes in attitude over time 
after graduation as well as changes occurring from matriculation to completion of the degree 
program. 
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□ Employer Surveys, Although employer surveys do not measure cognitive leaning, Evaluators agree 

that employer surveys can measure skill learning (behavioral learning). An example is an em- 
ployer survey that reflects how well a graduate has adapted to a given work environment, includ- 
ing how well the graduate fulfills the technical, organizational, or political demands made of him 
or her. 

Irrespective of the domain. Evaluators state that academic units need to expand the means they 
use to determine how much and what a student has learned. They suggest that if the necessary 
funds, time, incentives, and opportunity for faculty development in assessment techniques are 
made available, faculty will be able to develop and test additional ways to measure student 
learning. 



O. Evidence of Strong Institutional Conunitment and Support for Assessing Student Learning 

Evaluators identify a number of characteristics of institutional programs to assess student academic achieve- 
ment that are indicators of institutional commitment. 



Evidence of Strong Institutional Commitment to Assessment 

i> support from the governing board; 

i> provision of leadership of the assessment program by senior executive officers and aca- 
demic administrators at all levels; 

i> repeated public statements about assessment as an institutional priority, and the reasons 
for it, by the CEO and other senior officers; 

i> public expressions of personal and institutional commitment to assessment of student 
learning and use of its results by departmental chairs and other academic administrators; 

i> administrative use of professional development grants and other incentives, recognition, 
and rewards for faculty actively involved in assessment activities at the departmental, 
divisional, and institutional level; 

i> evidence of faculty knowledge about assessment of students’ academic achievement and 
the institution’s assessment program, and of wholehearted faculty participation in each 
academic unit’s assessment program; 

[> development of a standing committee, reporting directly to the CAO, on the assessment of 
student academic achievement; 

[> allocation of sufficient resources (personnel, space, equipment, and budget) to sustain on- 
going assessment efforts at the institutional and academic unit level; 

[> incorporation of responsibility for assessment into the position description of the CAO, 
and assignment of the authority and resources sufficient to be effective; 

i> integration of the assessment process into the planning and budgeting processes of the 
institution so that academic units’ approved plans for measuring student learning, costs 
associated with carrying them out, and subsequent changes the academic units wish to 
test to determine if they would increase student academic achievement, are routinely in- 
corporated into academic units’ plans and budget requests; 
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0 establishment of a line for assessment costs in the annual educational and general (E & G) 
operating budget; 

0 provision for external review of assessment efforts to ensure the best possible process 
and use of results and to establish a high level of credibility among university and commu- 
nity constituents. 



P. Evidence of Weak Institutional Commitment to Assessing Student Learning 

Evaluators identify several characteristics of programs ostensibly intended to assess student academic 
achievement that reflect weak institutional commitment to assessment. The following are examples they 
often cite: 



Evidence of Weak Institutional Commitment to Assessment 

i> lack of a campus-wide, systematic data collection process; 

i> no incentives, expectations, or “carrots and sticks” to foster faculty participation in all 
aspects of the program for the assessment of student academic achievement; 

0 no provision for a process by which ameliorative actions are to be tried when discrepan- 
cies are found between results of student academic achievement and faculty objectives for 
student learning; 

i> no process for documenting changes that have been made to improve student learning as 
a result of assessment efforts; 

0 no procedures set up to gather evidence as to whether the changes introduced correlate 
with actual improvements in student learning; 

0 no public documents, disseminated institution-wide, describing the program and explain- 
ing its value; 

0 no integration of the process and timetable for assessing student learning with those for 
planning and budgeting at the academic unit and institutional levels. 



Not surprisingly, next to asking for full and prompt implementation of the assessment program, the most 
frequent requests APR Panelists make of institutions preparing for their next comprehensive visit have to 
do with correcting the just-mentioned weaknesses. 



Conclusion 

This paper has summarized the favorable and critical comments of peer Consultant-Evaluators who have re- 
viewed the plans and programs for assessing student learning in a large, diverse sample of NCA member institu- 
tions. Evaluators without exception write favorably about institutions that have taken NCAs assessment initia- 
tive seriously. As professional educators they recognize that creating and sustaining a culture that supports 
ongoing efforts to improve students’ academic achievement is essential for strengthening American higher edu- 
cation. Their commitment is to assuring that students as individuals and as contributing citizens and members 
of the national labor force receive an education of the highest possible quality. 
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Evaluators are consistent in expressing their disappointment that a minority of institutions are still “planning to 
plan” for a program to assess student learning or reveal in other ways a low regard for the assessment of student 
academic achievement. There is now a large and compelling body of evidence that regular assessment of student 
academic achievement and the use of the results to improve student learning is an effective means of assisting 
institutions and their students. 

Evaluator comments have also shown that assessment does not necessarily require huge allocations of scarce 
resources, or necessitate eliminating all grass-roots assessment activities that may have arisen sporadically 
across an institution. To date, small or low-to-medium budget institutions have demonstrated that where there is 
institutional commitment to improving student learning, modest means and creativity have achieved assess- 
ment programs as effective as those in large and/or financially well-endowed institutions. Institutions need not 
discard their previous efforts, but can conduct incremental revisions and integration of assessment activities to 
reach their goals of implementing a useful program to assess student learning. Knowing that this information has 
been widely publicized within the academic community, team members find it difficult to understand how resis- 
tance toward assessment can persist in institutions committed to academic excellence and their own students’ 
optimum intellectual growth. 

As noted earlier, the Consultant-Evaluators’ judgments of the assessment programs they evaluated did not vary 
overall according to the type of institution where the Consultant-Evaluator holds appointment or the type of 
institution evaluated: public-private, size of student body, number of years, level or kinds of degrees-awarded, 
level of financial resources, or geographical location. This suggests that the peer Evaluators who are drawn 
from the nineteen-state NCA region, collectively and as individuals share an understanding of what constitutes 
good practice in programs for the assessment of student learning. It also reflects an underlying agreement on 
the meaning and value of assessment congruent with the following recently published definition of assessment 
drafted by the Director of the Assessment Forum at the American Association for Higher Education and refined 
by educators across the nation: 

Assessment is an ongoing process aimed at understanding and improving student learning. It 
involves making our expectations explicit and public; setting appropriate criteria and high stan- 
dards for learning quality; systematically gathering, analyzing, and interpreting evidence to de- 
termine how well performance matches those expectations and standards; and using the result- 
ing information to document, explain, and improve performance. When it is embedded effec- 
tively within larger institutional systems, assessment can help us focus our collective attention, 
examine our assumptions, and create a shared academic culture dedicated to assuring and im- 
proving the quality of higher education (AAHE Bulletin, November, 1995, p. 7). 

Significantly, as early as August 1993 NCA presaged the spirit of this proposed definition when it adopted the 
Commission Statement on Assessment of Student Academic Achievement that reads in part: 

...the Commission reaffirms its objective in this assessment initiative: to encourage in its institu- 
tions excellence in the teaching provided for students and in the learning achieved by them. 

While its is important that an institution respond fully and accurately to a variety of public de- 
mands for accountability, an institutional program for assessing student learning should emerge 
from a faculty and administrative commitment to excellent teaching and effective learning (Hand- 
book of Accreditation, 1994-96, p. 151). 

More recently, five months before the assessment definition appeared in the November 1995 AAHE Bulletin, the 
NCA expressed virtually identical sentiments in the following statement published in the June 1995 issue of the 
Briefing: 

the Commission is committed to the concept that assessment of student academic achievement 
is critical to the future health of its institutions and to the public perception of higher education 
nationally (p. 4). 

In the comprehensive visits taking place in 1995-96, Evaluation Teams have shifted their critical attention from 
assessment plans to the assessment programs based on those plans. Many NCA member institutions are already 
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using assessment in creative ways that result in benefits to students, faculty, and institutions that far exceed the 
demonstration of accountability mandated by national, regional, and state agencies, state coordinating or gov- 
erning boards, or institutional governing boards. Each institution that responds to the call for assessment of 
student academic achievement seizes the opportunity to more fully achieve its unique mission and educational 
goals. Each measurable improvement in student learning that results from using the findings of assessment to 
initiate change, increases the likelihood that graduates will reach their potential and the probability that each 
will be prepared to make the best contribution of which he or she is capable to family, community, and the future 
of this nation. 
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